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Abstract 

We consider the problem of fitting a linear model to data held by individuals who are concerned about their privacy. 
Incentivizing most players to truthfully report their data to the analyst constrains our design to mechanisms that 
provide a privacy guarantee to the participants; we use differential privacy to model individuals’ privacy losses. This 
immediately poses a problem, as differentially private computation of a linear model necessarily produces a biased 
estimation, and existing approaches to design mechanisms to elicit data from privacy-sensitive individuals do not 
generalize well to biased estimators. We overcome this challenge through an appropriate design of the computation 
and payment scheme. 
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1 Introduction 


Fitting a linear model is perhaps the most fundamental and basic learning task, with diverse applications from statistics 
to experimental sciences like medicine and sociology. In many settings, the data from which a model is to be learnt 
are not held by the analyst performing the regression task, but must be elicited from individuals. Such settings clearly 
include medical trials and census surveys, as well as mining online behavioral data, a practice currently happening at 
a massive scale. 

If data are held by self-interested individuals, it is not enough to simply run a regression—the data holders may 
wish to influence the outcome of the computation, either because they could beneht directly from certain outcomes, or 
to mask their input due to privacy concerns. In this case, it is necessary to model the utility functions of the individuals 
and to design mechanisms that provide proper incentives. Ideally, such mechanisms should still allow for accurate 
computation of the underlying regression. A tradeoff then emerges between the accuracy of the computation and the 
budget required to compensate participants. 

In this paper, we focus on the problem posed by data holders who are concerned with their privacy. Our approach 
can easily be generalized to handle individuals manipulating the computation’s outcome for other reasons, but for 
clarity we treat only privacy concerns. We consider a population of players, each holding private data, and an analyst 
who wishes to compute a linear model from their data. The analyst must design a mechanism (a computation he 
will do and payments he will give the players) that incentivizes the players to provide information that will allow for 
accurate computation, while minimizing the payments the analyst must make. 

We use a model of players’ costs for privacy that is based on the well-established notion of differential privacy 
Incentivizing most players to truthfully report their data to the analyst constrains our design to mechanisms that are 
differentially private. This immediately poses a problem, as differentially private computation of a linear model 
necessarily produces a biased estimation; existing approaches lll2|] to design mechanisms to elicit data from privacy- 
sensitive individuals do not generalize well to biased estimators. Overcoming this challenge, through appropriate 
design of the computation and payment scheme, is the main technical contribution of the present work. 


1.1 Our Results 

We study the above issues in the context of linear regression. We present a mechanism (Algorithm|2]l, which, under 
appropriate choice of parameters and fairly mild technical assumptions, satishes the following properties: it is (a) 
accurate (Theorem|4|i, i.e., computes an estimator whose squared L 2 distance to the true linear model goes to zero 
as the number of individuals increases, (b) asymptotically truthful (Theorem O, in that agents have no incentive to 
misreport their data, (c) it incentivizes participation (Theorem|3l, as players receive positive utility, and (d) it requires 
an asymptotically small budget (Theorem |6]l, as total payments to agents go to zero as the number of individuals 
increases. Our technical assumptions are on how individuals experience privacy losses and on the distribution from 
which these losses are drawn. Accuracy of the computation is attained by establishing that the algorithm provides dif¬ 
ferential privacy (Theorem|2|l, and that it provides payments such that the vast majority of individuals are incentivized 
to participate and to report truthfully (Theorems |3] and |5]l. An informal statement appears in Theorem[T] 

The fact that our total budget decreases in the number of individuals in the population is an effect of the approach 
we use to eliciting truthful participation, which is based on the peer prediction technology (Appendix lA.lb and of the 
model of agents’ costs for privacy (Section f2.41 i. A similar effect was seen by d. As they note, costs would no 
longer tend to zero if our model incorporated some hxed cost for interacting with each individual. 


1.2 Related Work 


Following 11131], a series of pap ers have studied data acquisition problems from agents that have privacy concerns. The 
vast majority of this work jl ll [mnn operates in a model where agents cannot lie about their priv ate information 
(their only recourse is to withhold it or perhaps to lie about their costs for privacy). A related thread 1 13 ,22, ^ explores 
cost models based on the notion of differential privacy jj^tj. 

Our setting is closest to, and inspired by, il2|] , who bring the technology of peer prediction to bear on the problem 
of incentivizing truthful reporting in the presence of privacy concerns. The peer prediction approach of Clll incen¬ 
tivizes truthful reporting (in the absence of privacy constraints) by rewarding players for reporting information that is 
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pred ictive of the reports of other agents. This allows the analyst to leverage correlations between players’ information. 
II 12 I] adapt the peer prediction approach to overcome a number of challenges presented by privacy-sensitive individu¬ 
als. The mechanism and analysis of lll2|] was for the simplest possible statistic—the sum of private binary types. In 
contrast, we regress a linear model over player data, a signihcantly more sophisticated learning task. In particular, to 
attain accurate, privacy-preserving linear regression, we deal with biased private estimators, which interferes with our 
ability to incentivize truth-telling, and hence to compute an accurate statistic. 

Linear regression under strategic agents has been studied in a variety of different contexts. Jst] consider an analyst 
that regresses a “consensus” model across data coming from multiple strategic agents; agents would like the consensus 
value to minimize a loss over their own data, and they show that, in this setting, empirical risk minimization is group- 
strategyproof. A similar result, albeit in a more restricted setting, is established by 13 ■ Regressing a linear model 
over data from strategic agents that can only manipulate their costs, but not their data, was studied by | ]_4| and (01, 
while lll6|] consider a setting without payments, in which agents receive a utility as a function of estimation accuracy. 
We depart from the above approaches by considering agents whose utilities depend on their loss of privacy, an aspect 
absent from the above works. 

Finally, we note a growing body of work on differentially private empirical risk minimization. Our mechanism is 
based on the outcome perturbation algorithm of fl. Other algorithms from this literature — such as the localization 
algorithm of (|T1] or objective perturbation of ||3l — could be used instead, and would likely yield even better accuracy 
guarantees. We chose the output perturbation mechanism because it provides an explicit characterization of the noise 
added to preserve privacy, which allows the analysis to better highlight the challenges of incorporating privacy into 
our setting. 


2 Model and Preliminaries 

We present our model and a technical preliminary in this section. A more detailed review of peer prediction, linear 
regression, and differential privacy can be found in AppendixiAl 

2.1 A Regression Setting 

We consider a population where each player i G [n] = {1,..., n} is associated with a vector Xi G (i.e., player 
features) and a variable yi G R (i.e., her response variable). We assume that responses are linearly related to the 
features; that is, there exists a 0 G such that 

yi = 9^Xi + Zi, foralHG[n], (1) 

where Zi are zero-mean noise variables. 

An analyst wishes to infer a linear model from the players’ data; that is, he wishes to estimate 9, e.g., by performing 
linear regression on the players’ data. However, players incur a privacy cost from revelation of their data and need to 
be properly incentivized to truthfully reveal it to the analyst. More specihcally, as in 0, we assume that player i can 
manipulate her responses yi but not her features Xi. This is indeed the case when features are measured directly by the 
analyst (e.g., are observed during a physical examination or are measured in a lab) or are verihable (e.g., features are 
extracted from a player’s medical record or are listed on her ID). A player may misreport her response yi, on the other 
hand, which is unverihable; this would be the case if, e.g., yi is the answer the player gives to a survey question about 
her preferences or habits. 

We assume that players are strategic and may lie either to increase the payment they extract from the analyst or to 
mitigate any privacy violation they incur by the disclosure of their data. To address such strategic behavior, the analyst 
will design a mechanism M. : (R.'^ x K)" —>■ x R" that takes as input all player data (namely, the features Xi 
and possibly perturbed responses yf), and outputs an estimate 9 and a set of non-negative payments to each 

player. Informally, we seek mechanisms that allow for accurate estimation of 9 while requiring only asymptotically 
small budget. In order to ensure accurate estimation of 9, we will require that our mechanism incentivizes truthful 
participation on the part of most players, which in turn will require that we provide an appropriate privacy guarantee. 
We discuss privacy in more detail in Section l23] Clearly, all of the above also depend on the players’ rational behavior 
and, in particular, their utilities; we formally present our model of player utilities in Section lX^ 
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Throughout our analysis, we assume that 6 is drawn independently from a known distribution T, the attribute 
vectors xt are drawn independently from the uniform distribution on the d-dimensional unit ballQ and the noise 
terms Zi are drawn independently from a known distribution Q. Thus 0, {x^}ie[n], and {zi}i^[n] ^re independent 
random variables, while responses {yi}ig[n] are determined by ([T]i. Note that as a result, responses are conditionally 
independent given 6. 

We require some additional bounded support assumptions on these distributions. In short, these boundedness 
assumptions are needed to ensure the sensitivity of mechanism Ai is finite; it is also natural in practice that both 
features and responses take values in a bounded domain. More precisely, we assume that the distribution A has 
bounded support, such that || 0||2 < B for some constant S; we also require the noise distribution Q to have mean zero, 
finite variance cr^, and bounded support; supp(C/) = [—M, M] for some constant M. These assumptions together 
imply that < B and \yi\ < B + M. 

2.2 Linear and Ridge Regression 

Let X = denote the n x d matrix of features, and y = £ R” the vector of responses. 

Estimating 9 through ridge regression amounts to minimizing the following regularized quadratic loss function: 

n n 

£{9; X,y)=Y, ^ x., y^) + T l|0|l^ (2) 

That is, the ridge regression estimator can be written as: 9^ = aigmiug^g^d C{9; X,y) = ( 7 / + X^X)~^X^y. 
The parameter 7 > 0, known as the regularization parameter, ensures that the loss function is strongly convex (see 
Appendix|0 and, in particular, that the minimizer of (O is unique. When 7 = 0, the estimator is the standard linear 
regression estimator, which we denote by 9^ = {X^X)~^X^y. The linear regression estimator is unbiased, i.e., 
under ([T]i, it satisfies E[0^] = 9. The same is not true when 7 > 0; the general ridge regression estimator 9^ is biased. 

2.3 Differential Privacy 

Recall the classic definition of differential privacy by j^: 

Definition 1 (Differential Privacy 1^). A mechanism M : D" TZ is e-differentially private if for every pair of 
databases Z?, D' £ D” differing only in one element, and for every subset of possible outputs 5 C 72., Pr[Al(77) £ 
5] <exp(e)Pr[7W(D') £5]. 

We depart from this classic definition, quantifying privacy violation instead through joint differential privacy 0 . 
Intuitively, full differential privacy requires that all outputs by the mechanism Ai, including the payment it allocates 
to a player, is insensitive to every player’s input. In settings like ours, however, it makes sense to assume that the 
payment to a player is also in some sense “private,” in that it is shared neither publicly nor with other players. To that 
end, we assume that the estimate 9 computed by the mechanism Ad is a publicly observable output; in contrast, each 
payment tt^ is observable only by player i. Hence, from the perspective of each player i, the mechanism output that is 
publicly released and that, in turn, might violate her privacy, is (9, ir-i), where 7 r_i comprises all payments excluding 
player i’s payment. 

Definition 2 (Joint Differential Privacy ifl^ l. Consider a mechanism Ai : D" O x 72”, for V, 0, 72 arbitrary 
sets. For each i £ [n], let (Ad (•))_, = (o, 7 r_i) £ (A x 72"“^ denote the portion of the mechanism’s output that 
is observable to outside observers and players j i. A mechanism Ad is e-jointly differentially private if for every 
player i, every database D £ 77”, every d' £ 77, and for every observable set of outcomes 5 C (A x 72”“^.' 

Pr [{Ai{D))_^ £ 5] < exp(e)Pr [(M(d',77_,))_, £ 5] . 

*See Theorem[7]and its accompanying Remai'k in Appendix lA.2l for a discussion of generalizing beyond the uniform distribution. 
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This relaxation of differential privacy is natural, but it is also necessary to incentivize truthfulness. Requiring that 
a player’s payment tt^ be e-differentially private implies that a player’s unilateral deviation changes the distribution of 
her payment only slightly. Hence, under full differential privacy, a player’s payment would remain roughly the same 
no matter what she reports, which intuitively cannot incentivize truthful reporting. 

We emphasize here that the existence of priors and the independence of responses are used only to prove the 
accuracy of the model learned and truthfulness, but not to ensure any privacy guarantee. Our mechanism satisfies 
joint differential privacy regardless of of whether the assumptions hold; if they do, accuracy and truthfulness follow. 
Further, the notion of e-joint differential privacy depends on both yi and Xi. although a player can only manipulate yi, 
both her response and her features are treated as “private” variables in our model, and both disclosures incur a privacy 
cost. Features should certainly be deemed private if, e.g., they are attributes in a player’s medical record, or outcomes 
of a medical examination. Moreover, ([T]) implies a correlation between features and the response, which can be strong, 
for example, in the case where 0 has small support; it is therefore reasonable to assume that, if the response is private, 
so should features correlated to this response. 


2.4 Player Utilities 


As discussed in the related work section, starting from ||13[] . a series of recent papers on strategic data revelation 
model player privacy costs as functions of the privacy parameter e. We also adopt this modeling assumption. Having 
introduced the notion of joint differential privacy, we now present our model of player utilities. We assume that every 
player is characterized by a cost parameter Ci S K+, determining her sensitivity to the privacy violation incurred by 
the revelation of her data to her analyst. In particular, each player has a privacy cost function fi{ci, e) that describes 
the cost she incurs when her data is used in an e-jointly differentially private computation. Players have quasilinear 
utilities, so if player i receives payment for her report, and experiences cost fi{ci,e) from her privacy loss, her 
utility is m = e). 

Following again recent work, we assume that fi can be an arbitrary function, bounded by an increasing monomial 
of e. In particular, we make the following assumption. 


Assumption 1. The privacy cost function of each player satisfies fi{ci, e) < Cif^ 


The monotonicity in e is intuitive, as smaller values imply stronger privacy properties, with e = 0 indicating the 
output is independent of player i’s data. We note that the quadratic bound in Assumption[T]was introduced by |0 and 
also adopted by d. As noted by the above authors, the quadratic bound can be shown to hold for a broad class of 
natural cost functions fp, we refer the reader to Appendix iDl for a formal description of this class. 

Throughout our analysis, we assume that the privacy cost parameters are also random variables, sampled from a 
distribution C. We allow a to depend on player Fs data {xi, yi)', however, we assume conditioned on {xi,yi), that a 
does not reveal any additional information about the costs or data of any other agents. Formally; 

Assumption 2. Given {xi,yi), (X-i,y-i,c-i) is conditionally independent of Ci: 

^^[{X-i,y-i,C-i)\{xi,yi),Ci] =Y'v[{X_i,y_i,C-i)\{xi,yi),c'^]for all {X_i,y_i,c-i), {xi,yi), a, c'. 


We also make the following additional technical assumption on the tail of C. 

Assumption 3. The conditional marginal distribution satisfies mina;._y. {Prcjr,^c\xi,yi[cj < f]) > 1 ~ for some 
constant p > 1. 

Note that Assumption|3]implies that Prar^^cici < t] > 1 — t~p. 


2.5 Mechanism Properties 

We seek mechanisms that satisfy the following properties: (a) truthful reporting is an equilibrium, (b) the estimator 
computed under truthful reporting is highly accurate, (c) players are ensured non-negative utilities from truthful re¬ 
porting, and (d) the budget required from the analyst to run the mechanism is small. We present here the standard 
definitions for these properties used in this paper. Consider a regression mechanism Ai. Let TTi{X,y) and be the 
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Algorithm 1 Truthful Regression Mechanism(a, b) 

Solicit reports X G and y £ R" 

Analyst computes 6^ = {X^X)~^X^y and 6^^ = {Xj^X-i)~^XZiy-i for each i £ [n] 
Output estimator 9^ 

Pay each player z, TTi = Ba,b{xJ9^i,xjE[e\xt,y^]) 


payment to player i when {X, y) is the collection of reports to the regression mechanism, and let fi{ci, e) be player i’s 
cost for participating in the mechanism. We dehne a strategy prohle a = (cti ,..., ct„) to be a collection of strategies 
(Ti (one for each player), mapping from realized data {xi,yi) to reports iji. Under strategy a player who has data 
{xi,yi) would report iji = ai{xi, yi) to the regression mechanism. 

Definition 3 (Bayes Nash equilibrium). A strategy profile a forms an ry-approximate Bayes Nash equilibrium if for 
every player i,for all realizable {xi,yi), and for every misreport yi yt. 


E[TTi{X,a{X,y))\ - fi{ci,e) > E[7ri(X, (j/i, cr_i(X_i, y_i)))] - f{ci,e) - y. 

Definition 4 (Accuracy). A regression is n-accurate if for all realizable parameters 9, it outputs an estimate 9 such 
thatE[\\9 - 9\\l] <ry. 

Definition 5 (Individually Rational). A mechanism is individually rational (IR) if E[TTi{X,y)] — fi{ci,e) > Q for 
every player i and for all realizable {X, y). 

We will also be concerned with the total amount spent by the analyst in the mechanism. The budget B of 
mechanism is the sum of all payments made to players. That is, B = tt^. 

Definition 6 (Asymptotically small budget). An asymptotically small budget is such that B = v) — o(l)) 

for all realizable {X, y). 


3 Truthful Regression without Privacy Constraints 


To illustrate the ideas we use in the rest of the paper, we present in this section a mechanism which incentivizes truthful 
reporting in the absence of privacy concerns. If the players do not have privacy concerns (i.e., Ci = 0 for all i £ 
the analyst can simply collect data, estimate 9 using linear regression, and compensate players using the following 
scoring rulell 

Ba.bip, q) =a-b{p-2pq + q^) ■ 

The mechanism is formally presented in Algorithm[T] In the spirit of peer prediction, a player’s payment depends on 
how well her reported jji agrees with the predicted value of yi, as constructed by the estimate 9^^of 9 produced by all 
her peers. We now show that truthful reporting is a Bayes Nash equilibrium. 

Lemma 1 (Truthfulness). For all a,b > 0, truthful reporting is a Bayes Nash equilibrium under Algo rithm\J\ 

Proof Recall that conditioned on Xi,yi, the distribution of X-i,y-i is independent of c^. Hence, assuming all other 
players are truthful, player z’s expected payment conditioned on her data {xi,yi) and her cost Ci, for reporting yi is. 


E\'Ki\xi,yi,Ci] = E Ba,bixJ9B,xjE[9\xi,yi])\xi,yi = Ba,b {xjE[9^f\Xi,yi],xjE[9\xi,yi^ . 


The second inequality is due to the linearity of Ba,b in its hrst argument, as well as the linearity of the inner product. 
Note that Ba,b is uniquely maximized by reporting iji such that E\9\xi,yi]^ Xi = E[9^^\xi,yi]^ Xi. Since 9^ is an 
unbiased estimator of 9, then E[9^^\xi,yi] = E\9\xi,yi]. Thus the optimal misreport is iji such that E\9\xi,yiY Xi — 
E[0|xi, yi\^ Xi, so truthful reporting is a Bayes Nash equilibrium. □ 

^This is a variant of the well-known Brier scoring rule H. See Appendix IA. 1 I for more details. 


5 








We note that truthfulness is essentially a consequence of (1) the fact that Ba^b is a strictly proper scoring rule (as 
it is positive-affine in its hrst argument and strictly concave in its second argument), and ( 2 ) most importantly, the 
fact that 9^^ is an unbiased estimator of 9. Moreover, as in the case of the simple peer prediction setting presented in 
Appendix lA.ll truthfulness persists even if 9^^ in Algorithm[T]is replaced by a linear regression estimator constructed 
over responses restricted to an arbitrary set 5 C [n] \ i. 

Truthful reports enable accurate computation of the estimator with high probability, with accuracy parameter 

9 = 0 {^). 

Lemma 2 (Accuracy). Under truthful reporting, with probability at least 1 — and when n > C(|)^(c? + 2) log d, 
the accuracy the estimator 9^ in Algorithm\T\is E 




Proof. Note that E 

of matrix X^X can 
lemma follows. 


9^-9 


= trace(Cov(0^)) @ cr^ trace . For i.i.d. features Xi, the spectrum 


be asymptotically characterized by a theorem of 


(see Theorem|2]in Appendix lA.2b . and the 

□ 


Remark Note that individual rationality and a small budget can be trivially attained in the absence of privacy costs. 
To ensure individual rationality of Algorithm[T] payments must be non-negative, but can be made arbitrarily small. 
Thus payments can be scaled down to reduce the analyst’s total budget. For example, setting a = b{B + 2B{B + 
M) + {B + MY — 1) and b — Yg ensures tt^ > 0 for all players i, and the total required budget is ^{2B + 4i?(f3 -f 
M) + {B + Mf) = 0{^). 


4 Truthful Regression with Privacy Constraints 

As we saw in the previous section, in the absence of privacy concerns, it is possible to devise payments that incentivize 
truthful reporting. These payments compensate players based on how well their report agrees with a response predicted 
by 9^ estimated using other player’s reports. 

Players whose utilities depend on privacy raise several challenges. Recall that the parameters estimated by the 
analyst, and the payments made to players, need to satisfy joint differential privacy, and hence any estimate of 9 
revealed publicly by the analyst or used in a payment must be e-differentially private. Unfortunately, the sensitivity of 
the linear regression estimator 9^ to changes in the input data is, in general, unbounded. As a result, it is not possible 
to construct a non-trivial differentially private version of 9^ by, e.g., adding noise to its output. 

In contrast, differentially private versions of regularized estimators like the ridge regression estimator 9^ can be 
constructed. Recent techniques have been developed for precisely this purpos^ not only for ridge regression but for 
the broader class of learning through (convex) empirical risk minimization IHot] - In short, the techniques by jst] and 
ifUl succeed precisely because, for 7 > 0, the regularized loss (|2]) is strongly convex. This implies that the sensitivity 
of 9^ is bounded, and a differentially private version of 9^ can be constructed by adding noise of appropriate variance 
or though alternative techniques such as objective perturbation. 

The above suggest that a possible approach to constructing a truthful, accurate mechanism in the presence of 
privacy-conscious players is to modify Algorithm [T] by replacing 9^ with a ridge regression estimator 9^, both with 
respect to the estimate released globally and to any estimates used in computing payments. Unfortunately, such an 
approach breaks truthfulness because 9^ is a biased estimator. The linear regression estimator 9^ ensured that the 
scoring rule Ba,b was maximized precisely when players reported their response variable truthfully. However, in the 
presence of an expected bias b, it can easily be seen that the optimal report of player i deviates from truthful reporting 
by a quantity proportional to h^Xi. 

We address this issue for large n using again the concentration result by lE^ (see Appendix IA.21 i. This ensures 
that for large n, the spectrum of X^X should grow roughly linearly with n, with high probability. By Q, this implies 
that as long as 7 grows more slowly than n, the bias term of 9^ converges to zero, with high probability. Together, 
these statements ensure that for an appropriate choice of 7 , we attain approximate truthfulness for large n, while also 
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ensuring that the output of our mechanism remains differentially private for all n. We formalize this intuition by prov¬ 
ing that our mechanism presented in Section l4n based on ridge regression, indeed attains approximate truthfulness 
for large n, while also remaining jointly differentially private. 

4.1 Private Regression Mechanism 

We present our mechanism for private and truthful regression in Algorithmic] which is a privatized version of Algo¬ 
rithm [T] We incorporate into our mechanism the Output Perturbation algorithm from |3, which first computes the 
ridge regression estimator and then adds noise to the output. This approach is used to ensure that the mechanism’s 
output satisfies joint differential privacy. 

The noise vector v will be drawn according to the following distribution Pl, which is a high-dimensional Laplace 
distribution with parameter Pl{v) oc exp f 11^112 )■ 


Algorithm 2 Private Regression Mechanism( 7 , e, a, b) 

Solicit reports X S ^ and y G R" 

Randomly partition players into two groups, with respective data parrs (ATq, yo) and (Xi, yi) 
Analyst computes 9^ = ( 7 / -I- X)~^y and 9^ = ( 7 / + XjXj)~^Xjyj for j = 0,1 

Independently draw v,vq,vi G R'^ according to distribution Pl 
Compute estimators 9^ — 9^ + v, 9 q = 9q + vq, and = 9^ + vi 
Output estimator 9^ 

Pay each player i in group j, tt^ = Xi, ¥\9\xi,yiYXi) for J = 1 


Here we state an informal version of our main result. The formal version of this result is stated in Corollary [T] 
which aggregates and instantiates Theorems |2|[3l|4||5l and| 6 | 

Theorem 1 (Main result (Informal)). Under Assumptions\I\^ flnc/|5] there exists ways to set 7 , e, a, and b in Algorithm 
| 2 |fo ensure that with high probability: 

1. the output of Algorithm^is o{-^)-jointly differentially private, 

2. it is an o -approximate Bayes Nash equilibrium for a (1 — o{l))-fraction of players to truthfully report their 

data, 

3. the computed estimator 9^ is o{l)-accurate, 

4. it is individually rational for a (1 — o{l))-fraction of players to participate in the mechanism, and 

5. the required budget from the analyst is o(l). 


5 Analysis of Algorithm |2] 

In this section, we flesh out the claims made in Theorem [T] Due to space constraints, all proofs are deferred to 
Appendix IbI 

Theorem 2 (Privacy). The mechanism in Algorithm^^is 2e-jointly differentially private. 

Proof idea We first show that the estimators 9^, 9q , together satisfy 2e-differential privacy, by bounding the 
maximum amount that any player’s report can affect the estimators. We then use the Billboard Lemma (Lemma|5]in 
Appendix IA. 3 I 1 to show that the estimators, together with the vector of payments, satisfy 2e-joint differential privacy. 

Once we have a privacy guarantee, we can build on this to get truthful participation and hence accuracy. To do so, 
we first show that a symmetric threshold strategy equilibrium exists, in which all agents with cost parameter Ci below 
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some threshold r should participate and truthfully report their yi. We define to be the cost threshold such that (1) 
with probability 1 — /3 (with respect to the prior from which costs are drawn), at least a (1 — a)-fraction of players 
have cost parameter Ci < and (2) conditioned on her own data, each player i believes that with probability 1 — a, 
any other player j will have cost parameter cj < 

Definition 7 (Threshold Ta^p). Fix a marginal cost distribution C on {ci}, and let 

= inf (l^r [|{* : Ci < r}| > (1 - a)n] > 1 - /?) , 


= inf ( min ( Pr [c, < r] ) > 1 — a 
-r \xi,yi \cjr.C\xi,yi J 

Define Ta^p to be the larger of these thresholds: Ta,p = max{r^ t „}. 

We also define the threshold strategy Or , in which a player reports truthfully if her cost Ci is below r, and is allowed 
to misreport arbitrarily if her cost is above r. 

Definition 8 (Threshold strategy). Define the threshold strategy CTt as follows: 


Report yi = yi ifci<T, 

Report arbitrary iji otherwise. 


We show that ^ forms a symmetric threshold strategy equilibrium in the Private Regression Mechanism of 
Algorithmic 

Theorem 3 (Truthfulness). Fix a participation goal 1 — a, a privacy parameter e, a desired confidence parameter j3, 
^ G (0,1), and t > 1. Then under Assumptions\I]and^ with probability \ — df and when n > C{j)'^{d + 2) logd, 
the symmetric threshold strategy cr-r^ ^ is an rj-approximate Bayes-Nash equilibrium in Algorithm^\for 


Tj = b 
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Proof idea There are three primary sources of error which cause the estimator 0^ to differ from a player’s posterior 
on 9. First, ridge regression is a biased estimation technique; second. Algorithm |2] adds noise to preserve privacy; 
third, players with cost parameter Ci above threshold Ta^p are allowed to misreport their data. We show how to control 
the effects of these three sources of error, so that 9^ is “not too far” from a player’s posterior on 9. Finally, we use 
strong convexity of the payment rule to show that any player’s payment from misreporting is at most p greater than 
from truthful reporting. 

Theorem 4 (Accuracy). Fix a participation goal 1 — a, a privacy parameter e, a desired confidence parameter /3, 
^ G (0,1), and t > 1. Then under the symmetric threshold strategy Algorithm^will output an estimator 9^ 

such that with probability at least 1 — /3 — , and when n > + 2) log d. 


Ellis" - S|0 = o +1)% (2)’ + (1)% 22 +1) . 

\\7 7e/ \ n / \^/ 7 I 


Proof idea As in Theorem[C we control the three sources of error in the estimator 9^ — the bias of ridge regression, 
the noise added to preserve privacy, and the error due to some players misreporting their data — this time measuring 
distance with respect to the expected L 2 norm difference. 

We next see that players whose cost parameters are below the threshold Tq -^3 are incentivized to participate. 
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Theorem 5 (Individual Rationality). Under Assumption\J] the mechanism in Algorithm^is individually rational for 
all players with cost parameters Ci < Ta,p as long as. 


a > ( —(4_B + 2M) H-—-——-^ ^ ] (^4" 26_B) + bB^ + Ta 

\1 7+(l-6d+2^ J 

regardless of the reports from players with cost coefficients above Ta,p. 

Proof idea A player’s utility from participating in the mechanism is her payment minus her privacy cost. The 
parameter a in the payment rule is a constant offset that shifts each player’s payment. We lower bound the minimum 
payment from Algorithmic and upper bound the privacy cost of any player with cost coefficient below threshold 
If a is larger than the difference between these two terms, then any player with cost coefficient below threshold will 
receive non-negative utility. 

Finally, we analyze the total cost to the analyst for running the mechanism. 

Theorem 6 (Budget). The total budget required by the analyst to run Algorithm \2} when players utilize threshold 
equilibrium strategy (Tt-q ^ A 


B<n 


a + 


— {4B + 2M) + 
7 


yB 

1 + 



{b + 2bB) 


Proof idea The analyst’s budget is the sum of all payments made to players in the mechanism. We upper bound the 
maximum payment to any player, and the total budget required is at most n times this maximum payment. 


5.1 Formal Statement of Main Result 

In this section, we present our main result. Corollary [T] which instantiates Theorems |2][3]|4]|5] and |6] with a setting of 
all parameters to get the bounds promised in TheoremU] Before stating our main result, we first require the following 
lemma which asymptotically bounds for an arbitrary bounded distribution. We use this to control the asymptotic 
behavior of Ta,p under Assumption^ 

Lemma 3. For a cost distribution C with conditional marginal CDF lower bounded by some function F: 


min ( Pr [c,- < rl ) > F(t), 

then 

Ta,p < max{F“^(l — a/3),F“^(l — a)}. 

We note that under Assumption[3l Lemma [3] implies that Ta,p < max{(a/3)“^/P, {a)~^^P}. Using this fact, we 
can state a formal version of our main result. 

Corollary 1 (Mainresult (Formal)). Choose 5 S (0, 2 + 2 p )• under Assumptions\I\^ and\^ setting 7 = 
e = a = (6B + 2M)(1 + B)‘^n~i + and b = n~i in Algorithm^ and taking a = n~^, /3 = 

„-f+5(i+p)^ ^ = 1/2, andt= ^c{d+ 2 )\o^ ’ ensures that with probability 1 — — n 2 +^( 1 +?).- 

7. the output of Algorithm^is O -jointly differentially private, 

2. it is an O ^n~ -approximate Bayes Nash equilibrium for a \ — O {n~^) fraction of players to truthfully 
report their data, 

3. the computed estimate 9^ is O {n~^^-accurate. 
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4. it is individually rational for a 1 — O (n fraction of players to participate in the mechanism, and 

5. the required budget from the analyst is O . 

This follows from instantiating Theorems |2][3 |4l|5] and|6]with the specified parameters. Note that the choice of S 
controls the trade-off between approximation factors for the desired properties. 

Remark Note that different settings of parameters can be used to yield a different trade-off between approximation 
factors in the above result. For example, if the analyst is willing to supply a higher budget (say constant or increasing 
with n), he could improve on the accuracy guarantee. 
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A Technical Preliminaries 


A.l Peer Prediction and the Brier Scoring Rule 


Peer prediction 112 111 is a useful method of inducing truthful reporting among players that hold data generated by the 
same statistical model. In short, each player reports her data to an analyst and is paid based on how well her report 
predicts the report of other players; tying each player’s payment to how closely it predicts peer reports is precisely 
what induces truthfulness. il2|] illustrate these ideas in the context of privacy-sensitive individuals through the use of 
the Brier scoring rule as a payment scheme among players holding a random bit. As we make use of the same 
technique, we review here how the Brier scoring rule can be used for basic peer prediction. 
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The basic Brier scoring rule was designed for the prediction of a binary event. Let I be an indicator of the event 
occurring. Then the payment for reporting that the event will occur with probability q is, 


BasicBrier{I, q) = 2Iq + 2(1 — /)(1 — q) — q'^ — {1 — q)'^. 

Following lfl3] . we define an extension of the basic Brier scoring rule. For any p and q, we define the payment function 
B{p, q) as follows; 

B(p, q) = l-2{p- 2pq + q^) 

Note that for the prediction of a binary event, B{p,q) is the expected payment according to BasicBrier{I,q) 
when the event will occur with probability p and the agent submits prediction probability q. That is, B{p,q) = 
V.ir^p[BasicBrier{I, q)]. By design, B{p, q) is a strictly proper scoring rule, which means it is uniquely maximized 
by a player truthful reporting her belief q about the probability of the event occurring. 

Algorithms [T] and |2] use payment rule Ba,bip, q), which is a parametrized rescaling of the scoring rule B{p, q), 
defined as follows: 

Ba,b{p, q) = a-b{p- 2pq + q^) . 

Any positive-affine transformation of a strictly proper scoring rule remains strictly proper The rescaled Brier 
scoring rule satisfies this criterion as Ba,b{p,q) = a' -f b'B{p,q) where a' = a — b/2 and b' = 6/2 > 0. Thus 
Ba,bip, q) is a strictly proper scoring rule, and is uniquely maximized by reporting the true probability q = p. 

For concreteness, we now provide an example to demonstrate how the payment rule B{p, q) can be used in peer 
prediction to truthfully elicit players’ beliefs. Consider a set of n players, each holding a binary variable bi G {0,1}. 
Assume that each of these variables is generated by independent Bernouli trials with parameter p, i.e., Pr(6i = 1) = p, 
for every i G [n]. We assume here that p is itself a random variable generated from a known prior over [0,1]. Each 
player reports a bit 6^ G {0,1} to the analyst, who wishes to estimate p as i analyst therefore wishes 

to incentivize truthful reporting of the bits bi, through an appropriate payment scheme. 

Let E[p I 6] be expected value of p conditioned on observing that a player’s bit is 6 G {0,1}. Put differently, 
for every player whose bit is 6, E[p | 6] captures her belief about the realization of p after she observes her own bit. 
Consider the following payment rule. To generate the payment for player i, the analyst selects a player j uniformly at 
random from [n] \ % and pays player v. 

B{b,,E[p\bi\) (3) 

Lemma 4. i^/ Under payments 01, truthful reporting is a Bayes-Nash equilibrium. 


Proof. Observe that for all q, q' G [0,1], B{q', q) is positive, so payments 0 are individually rational. Moreover, for 
all q' G [0,1], B{q', q) is a strictly concave function of q maximized at q' = q. Moreover, B{q', q) is an affine function 
of q'. If player Fs bit is bi and all other players report their bits truthfully (i.e., bj = bj for all j i), then player Fs 


expected payment is E 


B{bj,E[p I 6,]) I 6 ,] = B (e[6 , I 6,],E[p | 6,]) = B (e[p \ 6,],E[p | 6,]) . 


Hence, player Fs 


payment is maximized when bi = bi. 


□ 


Informally, the payment scheme 0 induces truthfulness by awarding a player the highest payment if the belief 
induced on p by her reported bit “agrees” with the belief induced by the bit of a random peer. We note that instead 
of the bit of a peer selected at random, any quantity whose expectation conditioned on bi would be equal to E[p | bi] 
would work as input to the payment rule. For example, using the average value 6s = S' C [n] \ i 

as the first argument of B would also induce truthful reporting. 


A.2 Properties of ridge regression 

As mentioned in Section the ridge regression estimator 9^ is biased, while the linear regression estimator 9^ is 
unbiased. Nevertheless, in practice 9^ is preferable to 9^ as it can achieve a desirable trade-off between bias and 
variance. In particular, consider the square loss error of the estimation 9^, namely, E[||6^ — If we condition on 
the true parameter vector 9 and the features X, this can be written as 

E[||6« - 9\\l] = E[||6« - E[6«]||| + ||E[0«] - 9\\l = trace(Cov(0«)) + || bias(0«)||^ (4) 
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where Cov(0'^) = E[(0^ —E[0'^])(0^—E[0^])^] andbias(0^) = E[0'^] — 0 are the covariance and bias, respectively, 
of estimator 6^. Assuming that the responses y follow ([T]0, then conditioned on X and 9, these can be computed in 
closed form as: 


cov( 0 ") = a\-fi + x' x)-^x ' a: (7/ + x' xy 


bias(0") = -7(7/ + a: ' X)-^9, 


(5) 


where cr^ is the variance of the noise variables Zi in ([T]i. It is easy to see that decreasing 7 decreases the bias, but may 
significantly increase the variance. For example in the case where rank(X) < d, the matrix A'^Ai is not invertible, 
and the trace of the covariance tends to infinity as 7 tends to zero. 

Whether trace(Cov(0^)) is large and, therefore, whether regularizing the square loss is necessary, depends on 
largest eigenvalue (i.e., the spectral norm) of (X^AT)”^. Although this can be infinite for arbitrary X, if the x^’s 
are drawn i.i.d. we expect that as n increases we will get estimates of lower variance. Indeed, by the law of large 
numbers, we expect that if we sample the features Xi independently from an isotropic distribution, then —{X^X) 
should converge to the covariance of this distribution (namely 'S = cl for some constant c). As such, for large n both 
the largest and smallest eigenvalues of X^X should be of the order of n, leading to an estimation of ever decreasing 
variance even when 7 = 0 . The following theorem, which follows as a corollary of a result by ll25ll (see AnnendixO. 
formalizes this notion, providing bounds on both the largest and smallest eigenvalue of X^X and 7/ + X^X. 

Theorem 7. Let ^ S (0,1), and f > 1. Let || • || denote the spectral norm. If {xi\i^\ji\ i.i.d. and sampled uniformly 
from the unit ball, then with probability at least 1 — d~^ , when n > C'(|)^(d + 2) log d, for some absolute constant 


C, then, 


x^atII < (1 + Ot^”' < 


7 / + a:'a: <7 + (i + 0 


d + 2 
1 


d + 2 


n, and || (7/ + A'^AT)”^ 


< 


and 


1 


T+(l-0d+2' 


Remark A generalization of Theorem |7] holds for Ixil ipu j sampled from any distribution with a covariance E 
whose smallest eigenvalue is bounded away from zero (see 11251] 1. We restrict our attention to the unit ball for simplicity 
and concreteness. 


A. 3 The Billboard Lemma 

A very useful result regarding jointly differentially private mechanisms that we use in our analysis is the so-called 
“billboard-lemma”: 

Lemma 5 (Billboard Lemma ifisll ). Let M. : 2?" O be an e-differentially private mechanism. Consider a set of 
n functions hi : D x O ^ TZ, for i £ [n]. Then, the mechanism M' : 2?" O x TZ+ that computes r = M.{D) 
and outputs A4'(D) = (r, hi(Il 2 D, r),..., hn{IlnD, r)), where Hi is the projection to player i’s data, is e-jointly 
differentially private. 

In short, the billboard lemma implies that if we can construct payments such that the payment to player i depends 
only on her data (e.g. Xi, yi) and a universally observable output that is e-differentially private (e.g., 6), then the 
resulting mechanism will be e-jointly differentially private. 

B Proofs from Section |5] 

B. 1 Proof of Theorem |2] (Privacy) 

We will now prove that the estimator 9^ and the vector of payments tt of the mechanism in Algorithm|2]is 2e-jointly 
differentially private. First, we need the following lemma to bound the sensitivity of 9^, formally defined in Definition 
|9l which is the maximum change in the output when a single player misreports her data. For vector-valued outputs, 
we measure this change with respect to the L 2 norm. 

^i.e., under truthful reporting. 
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Definition 9 (Sensitivity). The sensitivity of a function f : T) ^ TZ is the maximum L 2 norm of the function’s output, 
when a single player changes her input: 


Sensitivity of f = max \\f{D) — f(D ')\\2 

D,D', neighbors 

The following lemma follows from ||3l; a proof is provided for completeness. 

Lemma 6. The sensitivity of 9^ is i(4i? + 2M). 

Proof Let {X, y) and {X', y') be two arbitrary neighboring databases that differ only in the *-th entry. Let 6^ and 
[O^y respectively denote the ridge regression estimators computed on {X,y) and {X',y'). Define g{9) to be the 
change in loss when 9 is used as an estimator for {X', y') and {X, y). 


g{9)=C{9;Xyy')-C{9;X,y) 

= {9^x^ -yyf - {9^ x'i - y'yf 


Lemma 7 of Ist] says that if C{9] X, y) and £(0; X', y') are both L-strongly convex, then 9^ — {9^)' is bounded 

2 

above by i • maxg ||Vp( 0 )|| 2 . By Lemma fOl (in Appendix |E|i, both £{9;X,y) and C{9\ X',y') are 27 -strongly 


convex, so 


qR _ (J)Ry 


- 27 


' maxg II Vp( 0 )|| 2 . We now bound || Vp( 0)||2 for an arbitrary 9. 


l|Vp(6»)||2 = 2 \\{9^Xi - yi)xi - {9^x'i - y')a^*||2 

< 4 \9^Xi - yi\ ||a:i||2 

< 4 {\9^Xi\ + \y^\) 

< A{2B + M) 


Since this bound holds for all 6 >, it must be the case that maxg ||Vp( 6>)||2 < 4(2i? + M) as well. Then by Lemma? of 


QR_(^§Ry <—(2B + M) =-{AB+ 2M). 

2 27 7 

Since {X, y) and {X', y') were two arbitrary neighboring databases, this bounds the sensitivity of the computation. 
Thus changing the input of one player can change the ridge regression estimator (with respect to the L 2 norm) by at 


most i(4i? + 2M). 


□ 


We now prove that the output of Algorithm|2]satisfies 2e-joint differential privacy. 


Theorem 2 (Privacy). The mechanism in Algorithm\2\is 2e-jointly differentially private. 

Proof We begin by showing that the estimator 9^ output by Algorithm|2]is e-differentially private. 

Let h denote the PDF of 9^ output by Algorithmic and v denote the PDF of the noise vector v. Let {X, y) and 
{X',y') be any two databases that differ only in the i-th entry, and let 9^ and (9^)' respectively denote the ridge 
regression estimators computed on these two databases. 

The output estimator 9^ is the sum of the ridge regression estimator 9^, and the noise vector u; the only random¬ 
ness in the choice of 9^ is the noise vector, because 9^ is computed deterministically on the data. Thus the probability 
that Algorithm |2] outputs a particular 9^ is equal to the probability that the noise vector is exactly the difference 
between 9^ and 9^. Fixing an arbitrary 9^, let v = 9^ — 9^ and v' = 9^ — (9^)'. Then, 


h{9P\{X,y)) 

h{9P\{X',y')) 


yv) 


= exp 


-76 


%B + AM 



= exp 


76 


SB + AM 



(6) 


By definition, 9^ 
triangle inequality. 


9 P -\- V = [9^)' + v'. Rearranging terms gives 9^ — {9^)' = v' — v. By Lemma| 6 ]and the 


l2< 



{9^y 


< -(AB + 2M) 
2 7 
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Plugging this into Equation (|6]l gives the desired inequality, 


h{eP\{X,y)) 

h{6P\{X',y')) 


< exp 


76 


1 


AB + 2M 7 


{AB + 2M) 


exp(e). 


Next, we show that the output (0^, 9^ , 9f, {7ri}ig[„]) of the mechanism satisfies joint differential privacy using 
the Billboard Lemma. The estimators 9q and Of are computed in the same way as 0^, so Of and Of each satisfy 
e-differential privacy. Since Of and Of are computed on disjoint subsets of the data, then by Theorem 4 of 
together they satisfy e-differential privacy. The estimator a player should use to compute her payments depends only 
on the partition of players, which is independent of the data because it is chosen uniformly at random. Thus by the 
Composition Theorem in |l9(], the estimators (0^, 0f,0f) together satisfy 2e-differential privacy. 

Each player’s payment is a function of only her private information — her report {xi,yi) and her group in 
the partition of players — and the 2e-differentially private vector of estimators {0^, Of, Of). Then by the Billboard 
Lemma|5] the output (0^, Of ,9f, {7i'i}ig[„]) of Algorithmic] satisfies 2e-joint differential privacy. □ 


B.2 Proof of Theorem [3] (Truthfulness) 


In order to show that ^ is an approximate Bayes-Nash equilibrium, we require the following three lemmas. Lemma 
Id bounds the expected number of players who will misreport under the strategy profile Lemma [8] bounds the 

norm of the expected difference of two estimators output by Algorithm|2]run on different datasets, as a function of the 
number of players whose data differs between the two datasets. Lemma |9]bounds the first two moments of the noise 
vector that is added to preserve privacy. 


Lemma 7. Under symmetric strategy profile each player expects that at most an a-fraction of other players 

will misreport, given Assumption^ 


Proof. Let S-i denote the set of players other than i who truthfully report under strategy From the perspective 

of player i, the cost coefficients of all other players are drawn independently from the posterior marginal distribu¬ 
tion C\xi,yi. By the definition of Ta,p, player i believes that each other player truthfully reports independently with 
probability at least 1 — a. Thus E[|5'_j| \xi,yj\> (1 —a)(n —1). □ 


Lemma 8. Let 0^ and [0^)' be the ridge regression estimators on two fixed databases that differ on the input of at 
most k players. Then 



{o^y 


< -{AB + 2M) 

2 7 


Proof. Since the two databases differ on the reports of at most k players, we can define a sequence of databases 
Dq, ..., Dk, that each differ from the previous database in the input of at most one player, and Dq is the input that 
generated 0^, and Dk is the input that generated (0^)'. Consider running Algorithm |2] on each database Dj in the 
sequence. For each Dj, let Of be the ridge regression estimator computed on Dj. Note that Of — 0^ and Of = [0^)'. 


0^ -(O^) 


< 


- Of 


r\K I nR nR , nR 

-hi +0j_ - ... - Ok-1 + W/c-i - 


R 


- of 


-f 


0f-0: 


...-f 


- - 


< k ■ max 

3 


nR 

^3 


For each j. Of and Of.i are the ridge regression estimators computed on databases that differ in the data of at most a 
single player. That means either the databases are the same, so Of — Of.i and their normed difference is 0, or they 
differ in the report of exactly one player. In the latter case. Lemma |6] bounds \\0f — Of^i II 2 above by f{AB + 2M) 
for each j, including the j which maximizes the normed difference. 
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Combining this fact with the above inequalities gives. 


§R 




k 

< -(45 + 2M). 

2 7 


□ 

Lemma 9. The noise vector v added in Algorithm^satisfies: E[u] = 0 and E[||u||f] = 2 ^^[Iklb] = 

4B+2M 

■ye 

Proof. For every v € there exists —v £ that is drawn with the same probability, because ||u ||2 = || — u|| 2 - 
Thus, 

E[u] = V Pr(n = v)dv = - {v -\ — v) Pr('(; = v)dv — 0. 

J V ^ J V 

The distribution of u is a high dimensional Laplacian with parameter mean zero. It follows immedi¬ 
ately that E[||u|||] = 2 andE[||u|| 2 ] = □ 

We now prove that symmetric threshold strategy ^ is an approximate Bayes-Nash equilibrium in Algorithmic 

Theorem 3 (Truthfulness). Fix a participation goal 1 — a, a privacy parameter e, a desired confidence parameter fd, 
^ £ (0, 1), and t > 1. Then under Assumptions\I\and^ with probability 1 — d* and when n > C{^)‘^(d -|- 2) logd, 
the symmetric threshold strategy ^ is an rj-approximate Bayes-Nash equilibrium in Algorithm\2\for 


Proof Suppose all players other than i are following strategy ^. Let player i be in group 1 — j, so she is paid 
according to the estimator computed on the data of group j. Let 6^ be the estimator output by Algorithm |2] on the 
reported data of group j under this strategy, and let {9^)' be the ridge regression estimator computed within Algorithm 
IC when all players in group j follow strategy cr,-^ ^. Let 9^ be the ridge regression estimator that would have been 
computed within Algorithm |2] if all players in group j had reported truthfully. For ease of notation, we will suppress 
the subscripts on the estimators for the remainder of the proof. 

We will show that p is an approximate Bayes-Nash equilibrium by bounding player i’s incentive to deviate. 
We assume that Ci < (otherwise there is nothing to show because player i would be allowed to submit an arbitrary 
report under ar^ p). We hrst compute the maximum amount that player i can increase her payment by misreporting to 
Algorithmic Consider the expected payment to player i from a fixed (deterministic) misreport, iji = yi + S. 


HBa,b{{d^V x„ E[9\xi,yi]^ Xi)\x„yi] - E[Ba,b{{0^V Xi,E[9\xi,yi\^ xyfxy, yy] 
= Ba,b{E[9^\xi, yi]^Xi, E[0|a;i, - Ba,b(E[9^\xi,yi]^ Xi,E[9\xi,yi]^ xf 


The rule Ba^b is a proper scoring rule, so it is uniquely maximized when its two arguments are equal. Thus any 
misreport of player i cannot yield payment greater than Ba^b{E[9^\xi, yi]^ Xi,E[9^\xi,yi]^ xf, so the expression of 
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interest is bounded above by the following. 


Ba,b{^0^\x^,y^YXi,Wfi^\xi,yiYXi) - Ba,b{^0^\x^,y^YXi,'E[9\xi,y^YXi) 

= a-b (E[§^\xi,y^]^x^ - 2{E[0^\xi,y^]^x^)^ + {E[e^\xi,y^]^ 

-a + b (E[e^\xi,yt]^Xt - 2{E[e^\xi,yi\^Xt){E[9\xi,yi\^xt) + {E[9\x^,yi]^xtf'^ 
= b (^(E[0^|a;i, yi]^Xi)'^ - 2{E[9^\xi, yi]^x^){E[9\xi,yi]^Xi) + {E[9\xi, y^]^XiY^ 

= b yi]^Xi - E[9\xi, yi\^Xi'j 

= b(E[9^-9\xi,y,]^x,y 
<b{\\E[9P-9\x.,y.]\\l\\xS) 

<b\\E[9P-9\x,,y,]\\l 


We continue by bounding the term ||E[0^ — 9\xi, yi] ||2. 

||E[0^ - 9\xu y^] II2 = ||E[r - 9^ + - 9\x,, y ,]\\2 

= ||E[(0«)' + v-9^ + 9^- 9\x,,y,]\\2 
= ||E[r;|x„ y,] + E[(0«)' - 9^\x,, y,] + E[0« - 9\x,, y,]h 
< ||E[r;|a;i,y,]||2 + ||E[(0^)' - 9^\xi,yi\\\2 + ||E[0^ - 9\xi,yi]\\2 


We again bound each term separately. In the first term, the noise vector is drawn independently of the data, so 
E[v\xi,yi\ = E['(;], which equals 0 by Lemma|9] Thus ||E['(;|a;i, yi] II 2 = 0. 

Jensen’s inequality bounds the second term above by E[|| [9^)' — 9^\\2\xi, y^]. The random variables {9^)' and 9^ 
are the ridge regression estimators of two (random) databases that differ only on the data of players who misr^orted 
under threshold strategy cr,-^ ^. By Lemma|7] player i believes that at most an players will misreport their y^O so for 
all pairs of databases over which the expectation is taken, {9^)' and 9^ differ in the input of at most an players. By 
LemmaO their normed difference is bounded above by ^(4i3 + 2M). Since this bound applied to every term over 
which the expectation is taken, it also bounds the expectation. 

For the third term, E[0^ — 9\xi, y^] = bias(0^|a:i, yi). Recall that 9^ is actually 9f, which is computed indepen¬ 
dently of player i’s data, but is still correlated with (xi, yi) through the common parameter 9. However, conditioned 
on the true 9, the bias of 9^ is independent of player Ts data. That is, bias(0^|a;i, yj, 9) = bias(0^|0). We now 
expand the third term using nested expectations. 


^x,z,e 

9^ - 9\xi,y^ 

= Eg 

- 0\xi,yi,9] 



= Eg 

bias(0^ Xi 

VrA) 




= Ee 

bias(0^|6*) 




= bias(0^) 

= -lill + X)-^9 

'^Lemma[7]proniises that at most ain — 1) players will misreport. We use the weaker bound of an for simplicity. 
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Then by Theorem|7] when n>C(|)^(c? + 2) log d, the following holds with probability at least 1 — d . 

m0^-e\x,,yi]\\2 = II -7(7/ + ^^^)-'0||2 
< 711(7/+ ^^^)-'ll2||0||2 

(7+(l-0d72^) ^ 

jB 

7 + (i-<^)aT2^ 

We will assume the above is true for the remainder of the proof, which will be the case except with probability 
.2 .2 

at most d~ . Thus with probability at least 1 — d~ , and when n is sufficiently large, the increase in payment from 
misreporting is bounded above by 

6||E[0^ - e\xi, y.]\\l < b (— (4B + 2M) + ^ ^ 

V7 7 + (l-C)d+2^ 

In addition to an increased payment, a player may also experience decreased privacy costs from misreporting. By 
Assumption[T] this decrease in privacy costs is bounded above by c^e^. We have assumed (otherwise player 

i is allowed to misreport arbitrarily under and there is nothing to show). Then the decrease in privacy costs for 

player i is bounded above by 

Therefore player Ts total incentive to deviate is bounded above by rj, and the symmetric threshold strategy cjr^ ^ 
forms an p-approximate Bayes Nash equilibrium for 

r^ = hi—{4B + 2M)+ 1 —) +Ta,/ 3 e^ 

V T' 7+ (1-^772^/ 

□ 



B.3 Proof of Theorem IH (Accuracy) 


In this section, we prove that the estimator 0^ output by Algorithm|2]has high accuracy. We first require the following 
lemma, which uses the concentration inequalities of Theorem [T] to give high probability bounds on the distance from 
the ridge regression estimator to the true parameter 0. 

Lemma 10. Let 0^ be the ridge regression estimator computed on a given database {X, y). Then with probability at 
least 1 — , as long as n > C{j)‘^{d + 2) logd 


E[||0«-0||2]< 


jB 


(l+^)dT2^ 


and 


7+(1-07T2” 


E[||0«-0||2]< 


(7 + (1-07T2'^)^ 


yB + Mn 


7+(1-C)7T2’ 


Proof. Recall from Section lA)2l that. 

E[|| 0 ^ - 6'||2] = II bias(0^)||2 + trace(Cov( 0 ^)), 


E[||0^ - 0 II 2 ] = E[\\0^ - E[0^] + E[0^] - 0 II 2 ] 

< E[||0« - E[0«]||2] + E[||E[0«] - 0 II 2 ] 
= E[||0«-E[0«]||2]+E[||bias(0«)||2] 
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We now expand the remaining terms: || bias(0'^)||2 and trace(Cov(0'^)) and E[||0^ — E[0^] II 2 ]. For the remain¬ 
der of the proof, we will assume the concentration inequalities in Theorem|7]hold, which will be the case, except with 
probability at most , as long as n > C(j)^(d + 2) logd. 


||bias(d«)||2 = ||-7(7/ + ^^^)-'0||2 
<7||0||2||(7/ + ^^^)-i2 
<jBH(jI + X^X)-^H2 
jB 

~ 7 + (^-0dh^ 


trace(Cov(0^)) = || Cov(d ^)||2 

= || ct 2 (^/ + x^x)-^x^x(-ri + x^x)-^\\l 

<a^UjI + X^X)-X\\X^X\\l\\{ 7 l + X^X)-X 


<a" 


<a" 


1 


,7+(l-0dT2’^ 


/ 


(1 + e) 


d + 2 


,7 + (l-0dT2' 


(l + e)dT2^ 

^(7 + (i-e)dT2’^) 


E[\\§^ - E[ 0 «]|| 2 ] = E[|| 0 « -{0 + bias( 0 «))|| 2 ] 

= E[|| ( 7 / + x^xy^x^y - 0 + ( 7 / + x^xyyiey] 

= e[||( 7 / + x^x)-^x^{x0 + z) - 0 + ( 7 / + x^xyyieh] 

= E[|| ( 7 / + X^X)-\X^X + 7/)0 - 0 + ( 7 / + x^xy^x^zh] 

= E[||0 -9 + yi + x^xy^x^zh] 

= E[\\yi + x^xy^x^zh] 

<E[\\yi + x^xy%\\x^zh] 

<E[\\yi + x^xy^\\2Mn] 

^ Mn 

~ 7+{^-0d^ri 


Using these bounds, we see: 


E[||0^ 


l^]< 


"fB 


7+ (1-0 


+ cr^ 


d-\-2‘ 


(l+0dT2^ 
_(7 + (i-0aT2O 


and 


E[||0^ 


2] < 


jB 


7 + (l-0dT2^ 
7 _B + Mn 
7 + (l-0dT2^ 


Mn 


7 + (l-0dT2’^ 


□ 
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We now prove the accuracy guarantee for the estimator 9^ output by Algorithmic 

Theorem 4 (Accuracy). Fix a participation goal 1 — a, a privacy parameter e, a desired confidence parameter /3, 
^ G (0,1), and t > 1. Then under the symmetric threshold strategy ar^, p, Algorithm^will output an estimator 9^ 
such that with probability at least 1 — 13 — , and when n > + 2) log d, 


E[||0^ 





Proof. Let the data held by players be (X, y), and let y = ?/ + ^ be the reports of players under the threshold strategy 
^. As in Theorem[C let 9^ be the estimator output by Algorithm|2]on the reported data under this strategy, and let 
[9^)' be the ridge regression estimator computed Algorithm |2] when all players follow strategy ar^ Let 9^ be the 
ridge regression estimator that would have been computed within Algorithm |2] if all players had reported truthfully. 
Recall that v is the noise vector added in Algorithm^ 


E[||0^ 


\i]=^\\9^ -9^ + 9^-9\\i 


= E 


\\o^+ 


<E[||6i^-6i"||^]+E[||6» 


9\\l + 2{9^ -9^p^ -9^^ 
- 9\\l]+2E\\\9P - 9’^U\9’^ 


Oh] 


We start by bounding the first term. Recall that the estimator 9^ is equal to the ridge regression estimator on the 
reported data, plus the noise vector v added by Algorithm^ 


E[||0^ - 0^111 =E[||(0«)' + u-0^111 

= E[||(0«)' - + E[||z;||2] + 2E[((0«)' - 9^,v)] 

= E[||(0«)' - 9^\\l] + E[||u||2] + 2(E[(0«)' - 0''],EH) 

= EIIK^-^)' - 9^\\\\ + 2 (byLemmaO 

The estimators {9^)' and 9^ are the ridge regression estimators of two (random) databases that differ only on the 
data of players who misreported under threshold strategy ^. The definition of ensures us that with probability 
1 — at most an players will misreport their yj. For the remainder of the proof, we will assume that at most an 
players misreported to the mechanism, which will be the case except with probability f3. 

Thus for all pairs of databases over which the expectation is taken, (6*^)' and 9^ differ in the input of at most an 

players, and by LemmaO their normed difference is bounded above by ^^(45 + 2M)^ . Since this bound applies 
to every term over which the expectation is taken, it also bounds the expectation. 

Thus the first term satisfies the following bound; 


an 


m\0 -0\\i]<{ —(4B + 2M) +2 


AB + 2M 

7e 


By LemmafTOl with probability at least 1 — d when n > C{jh{d + 2) logd, the second term is bounded above 


by 


E[||0«-0|||< 


yB 


h + O^ri 




il + P - 0dT2^)'^ , 


We will also assume for the remainder of the proof that the above bound holds, which will be the case except with 
,2 

probability at most d . 
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We now bound the third term. 


2E[||r - e^2\\o^ - dh] = 2E[||(0«)' + V - 9^h\\o^ - eu 
<2E[(||(0«)'-0«|U + ||r;||2) \\6^ - 6h] 

= 2E[||(0«)' - 0«||2||0^ - Oh] + 2E[||u||2||0^ - eu 
= 2E[||(0«)' - e^hWe^ - eh] + 2E[||?;||2]E[||0« - eh] (by independence) 

= 2E[||(0«)' - e^hh^ - Sh] + 2 - ^ 112 ] (by LemmalD 

We have assumed at most an players misreported (which will occur with probability at least 1 — jj), so for all 
pairs of databases over which the expectation in the first term is taken, Lemma^bounds ||(0^)^ — above by 
^(45 + 2M). Thus we continue bonding the third term: 


2E[\\ie^y-ep\2]]e^-eh] + 2 


AB + 2M 

7e 


E[||0«-0|h 


< 2E[f —(4B + 2M )) ||0« - eh] + 2^^-i^E[||0« - 6 »|| 2 ] (by Lemma[ 8 l) 


= 2 

= 2 

< 2 


V 7 
an 

7 

an 

7 

an 


7e 

(4S + 2M) ) E[|| 0 « - eh] + + - eh] 


(4S + 2M) 


AB + 2M 

7e 


76 

E[||0«-0||2] 


7 


(4B + 2M) 


AB + 2M\ -/B + Mn 


76 


7+(l-C)d+2’ 


(by LemmafTOb 


We can now plug these terms back in to get our final accuracy bound. Taking a union bound over the two failure 
probabilities, with probability at least 1 — /3 — , when n > (7(|)^(d + 2) log d: 


an 


E[\\e^-eh]< {—{4B + 2M)] +2 


AB + 2M 

76 


'jB 


J+(l-0d+2^, 


+ a^ 


+ ^)d7' 


_(7 + (l-0^«)^ 


„ / an AB + 2M \ aB + Mn 

+ 2 —{AB + 2M) + - ' ' 

7 7e 


7+(l-0dT2’" 


□ 


B.4 Proof of Theorems |5] and |6] (Individual Rationality and Budget) 

In this section we first characterize the conditions needed for individual rationality, and then compute the total budget 
required from the analyst to run the Private Regression Mechanism in Algorithmic Note that if we do not require 
individual rationality, it is easy to achieve a small budget: we can scale down payments as in the non-private mechanism 
from Section IC However, once players have privacy concerns, they will no longer accept an arbitrarily small positive 
payment; each player must be paid enough to compensate for her privacy loss. In order to incentivize players to 
participate in the mechanism, the analyst will have to ensure that players receive non-negative utility from participation. 

We first show that Algorithmic is individually rational for players with privacy costs below threshold. Note that 
because we allow cost parameters to be unbounded, it is not possible in general to ensure individual rationality for all 
players while maintaining a finite budget. 
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Theorem 5 (Individual Rationality). Under Assumption\J] the mechanism in Algorithm^is individually rational for 
all players with cost parameters Ci < Ta,p as long as. 


a > [ —(4_B + 2M) H- - -——^-1” ^ ] {b A- 2bB) + bB^ + Tq, 

regardless of the reports from players with cost coefficients above Ta,/ 3 . 

Proof Let player i have privacy cost parameter Ci < Ta^p, and consider player i’s utility from participating in the 
mechanism. Let player i be in group 1 — j, so she is paid according to the estimator computed on the data of group j. 
Let 9^ be the estimator output by Algorithm|2]on the reported data of group j under this strategy, and let be the 
ridge regression estimator computed within Algorithm|2]when all players in group j follow strategy ^. Let 6^ be 
the ridge regression estimator that would have been computed within Algorithm|2]if all players in group j had reported 
truthfully. For ease of notation, we will suppress the subscripts on the estimators for the remainder of the proof. 

E[ui{xt,yt, iji)] = E[Ba,b{{0^VXt,E[9\xi, Xt)\xt,yt] - E[fi{ci, e)] 

> x„E[e\x,,y,]^x,)\x„y,] - (by Assump.[B 

= Ba,biE[9^\x^,y^]^Xi,E[9\xi,y^]^Xi) - 

We proceed by bounding the inputs to the payment rule, and thus lower-bounding the payment player i receives. 
The second input satisfies the following bound. 

E[e\xi,yr]^Xi < ||E[0|a:i,yj]||2||a;j||2 < B 

We can also bound the first input to the payment rule as follows. 

E[9^\xi, yi]^Xi = y^]^x^ + E[v\x^,yiYx^ 

= E[{9^)'\xi,yi]^ Xi 

< || E [( 0 «)>„ j /,]|| 2 || a ..||2 

< ||E[(0«)' - e^\x^,yi \\\2 + ||E[0^ - e\x^,y ^]\\2 + ||E[0|:r„y ,]||2 

< -f 2M) H--=-h B (by Lemma[8]and Theoreml?]) 

7 7 + (l-0dT2” 

Recall that our Brier-based payment rule is Ba,b{p,y) = a — b (p — 2pq + q^), which is bounded below by 
a — b\p\ — 2b\p\ |g| — b\q\‘^ = a — \p\{b + 2b\q\) — b\q\'^. Using the bounds we just computed on the inputs to player 
Fs payment rule, her payment is at least 


I fJCIO , 

TTi > a— —(45 -I- 2M) + 


yB 


7 ' ' l + 


+ B {b + 2bB)-bB^. 


Thus her expected utility from participating in the mechanism is at least 

C olti "T B 

— (45 -f 2M) H-—-—— 

Player i will be ensured non-negative utility as long as, 

yB 


+ b] {b + 265) - 65^ - Ta,pe^. 


a> —(4B + 2M) + 


7 




+ 5 j (6 + 265) -f 65 -I- Ta,pe ■ 


□ 
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The next theorem characterizes the total budget required by the analyst to run Algorithmic 

Theorem 6 (Budget). The total budget required by the analyst to run Algorithm \2} when players utilize threshold 
equilibrium strategy CTtq p A 


B<n 


a + 


— {4B + 2M) + 
7 


yB 

7 + (l-0dT2« 



{h + 2hB) 


Proof. The total budget is the sum of payments to all players. 


B = ^E[7ri] = '^E[Ba,t{0^V x„E[9\xi,yi]^ Xi)\x„yi] 

i—1 i—1 

n 

= Ba,b{E[e^\xi,yi]^Xi,E[e\xt,yi]^Xt) 

Recall that our Brier-based payment rule is Ba^b{p, q) = a — b {p — 2pq + q^), which is bounded above by a + b\p\ + 
2b\p\ Igl = a + \p\{b + 26|(7|). Using the bounds computed in the proof of Theorem|5] each player i receives payment 
at most, 

TTi > a + [ — (4i3 -f 2M') H- - -——=-1” I {b A- 2bBf 

V 7+ (1-03^2’" / 

Thus the total budget is at most; 

B = YE[TTi]<n(a+ I —{4B + 2M) + -- + B] {b + 2bB) 

h V 7+(1-03T2« J 

□ 



B.5 Proof of Lemma |3] (Bound on threshold Tq, 

Lemma 3. For a cost distribution C with conditional marginal CDF lower bounded by some function F: 


then 

Proof. We first bound ^. 


min I Pr [c, < tI 

Xi.Vi \cj~C|xi,yi 


> nr), 


"ra./S < niax{F ^(1 —Q!/3),F ^(1 —a)}. 


rl,ii = inf (I^r [|{i : a < t}| > (1 - a)n\ > 1 - 

= inf ^ Pr [|{i : a > t}| < an] >1-/3^ 

= inf ^1 — Pr [|{i : Ci > t}| > an] >1-/3^ 

= inf ^ Pr [|{i : q > r}| > an] < /3^ 


We continue by upper bounding the inner term of the expression. 

Ellii • c > rfl 

Pr [|{i : Ci > t}| > an] < —^—— (by Markov’s inequality) 
c~c cyn 


n Pr[ci > t] 
an 

Pr[ci > t] 


(by independence of costs) 
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From this bound, if then also Prc,,^c [|{* ■ Ci > t}\ > an] < /3. Thus, 


inf ^ Pr [|{i : a > r}| > an] < < inf -ll < ^ 

since the infimum in the first expression is taken over a superset of the feasible region of the latter expression. Then, 

'Pr[c^ > t] 


T-a,/3 < inf 


<P 


= inf {Pr[ci >t]< a/3) 

T 

= inf (1 — Pr[ci < r] < a/3) 


= inf (C'(r) > 1 — a/3) 

r 

< inf (F(r) > 1 — a/3) 

r 


(since the extremal conditional marginal bounds the unconditioned marginal) 
= inf (r > — a^)) 

= F-i(l-a/3) 


Thus under our assumptions, p < F ^(1 — a/3). 

We now bound r^. 

= inf ^min {Pr^.^c\xi,yi[(^j < t]) > 1 - a^ 

< inf {F{t) > 1 — a) 

T 

= inf (t > — a)) 

= F-\l-a) 

Finally, 

Ta,i3 = max{T^_^,r^} < max{F“^(l - a/3),F“^(l - a)}. 


□ 


B.6 Proof of Corollary [T] (Main result) 

Corollary 1 (Main result (Formal)). Choose S € (0, 2 + 2 p ^' under Assumptions\]]^ and\^ setting a = n~^, 
/3 = e = 7 = n^~i, a = {6B + 2M)(1 + B)^n~i + b = n~i, ^ = 1/2, and 

t = 4 ^c(d+ 2 )\ogd Algorithin^ensures that with probability 1 — d®*-’ — n 2+'5 (i+p); 

7. the output of Algorithm^is O -jointly dijferentially private, 

2. it is an O ^n~ -approximate Bayes Nash equilibrium for a \ — O {n~^) fraction of players to truthfully 
report their data, 

3. the computed estimate 9^ is O {n~^^-accurate, 

4. it is individually rational for al — O {n~^) fraction of players to participate in the mechanism, and 

5. the required budget from the analyst is O ( J. 
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Proof. Choose 6 € (0, Note that this ensures 5 < 1/2. Let a = n ^ and /3 = '^(i+p) as we have chosen. 

By the constraint that J < 2 + 2 p ’ ensured that /3 = o(l). By Lemtna[2 Tq .,/3 < max{(a/3) = 

(a/3)“^/*’ since a, (3 = o(l) andp > 1. Then r „_/3 = O 


Setting ^ = 1/2 and t = we ensure that with probability 1 — d 4C(d+2)iogd = x — dP^ the 

bounds stated in Theorem|2]hold. With probability 1 — /3, at most an a-fraction of players will have cost parameters 
above Ta^p. Taking a union bound over these two failure probabilities, the bounds in Theorems |2] [314] |5] and|6]will 
all hold with probability at least 1 — . For the remainder of the proof, we will assume all bounds 

hold, which will happen with at least the probability specified above. 

First note that by Theorem |2] Algorithm |2] is 2e-jointly differentially private. By our choice of e, the privacy 
guarantee is 2n~^~^^ = o(yPi). 


Recall that by Theorem[3 it is a 


b (t^(4B + 2M) + 


_ jB 

T'+(l-4)dT2^ 


+ Ta,pe 


-approximate Bayes-Nash equi¬ 


librium for a (1 — a)-fraction of players to truthfully report their data. Taking B, M, and d to be constants, it is a 


0 b 


{7+iy 


+ Ta 


-approximate BNE. To achieve the desired truthfulness bound, we require (among other 


things) that Ta^pe^ = o(-). Given the bound on r^^p, it would suffice to have e = o(n 4 + 2 ). This is satisfied by 
our choice of e = n-^+^because 5 < 1/2. After setting 6 = o(i), we will have the desired truthfulness bound if 
^ = 0 ( 1 ). This implies the following constraints on 7: we require 7 = uj{na) = uj{v}~^) and 7 = o{n). 

Our choice of 7 = ‘n}~i satisfies these requirements. Due to our choice of 6 = the approximation factor will 

be dominated by r^^pe^ = O = o(l). Thus truthtelling is an O = o(l)-approximate Bayes-Nash 

equilibrium for all but an n~^ = o(l)-fraction of players. 

Recall from Theorem |4] that the estimator 9^ is O + ^ + -accurate. We 

have already established that ^ = o(l) and = o(l). Trivially, ^ = o(l). We turn now to the term For this 
term to be o(l), we require 7 = uj{^) = uj Our choice of 7 = ensures this requirement is satisfied. 

Since ^ ^ = o(l)^ then so must be = o(l). The accuracy bound will be dominated by three terms; 

first , second ^ and third = n~i. Thus, Algorithmic outputs an estimator with accuracy 


O = o(l)- 

Theorem|5]says that the mechanism in Algorithm|2]is individually rational for a (1 — a)-fraction of players as long 
as a > + 2M) + 1 ^ {b+2bB) + bB'^+Ta^pe^. We now expand each term of this expression 

to prove that our choice of a satisfies the desired bound. Consider the first term; ^{4B + 2M) = n~i {IB -f 2M). 
This term is decreasing in n, so it can be upper bounded by its value when n = l. Thus ^ (4i? + 2M) < AB + 2M. 
Now consider the second term; 


'jB 


n 


1- 




n 2 S 


7 + (1-07T 


:n 


1 _ A 


= B 1- 


1 


The final term 


^_,_2 " - -r 2(d-|-2)"' - T 2(d-|-2) 

is always negative, so the entire term 


2{d + 2)n-^ -f 1 J 
can be bounded above by B. We 


—3—j— is always negative, so the entire term , 1 — 

2(d-r2)n-5+l 7-t(l-4)^r 

can simplify the expression b -f 2bB -f bB^ as (1 + Bf^b = (1 -f B')^n~^l'^. Finally, as noted earlier (and due 
to to Lemma O, we can upper bound T^^pe^ < n“ 2 +^. Combining all of these bounds, it would suffice to set 
a > {6B + 2M){1 + B)^n~^/^ + . We set a to be exactly this bound. Then it is individually rational for a 

1 — a = 1 — n~^ = 1 — 0 ( 1 ) fraction of players to participate in the mechanism. 


By Theorem|6l the budget required from the analyst isS<n a + {— (45 -f 2 M) H— 1 -h B 


From our choice of a = 0 


{n 2 +^^ 


') {h + 265) 


O {n{n 2 +n 2 +'^)^ = O 2+‘5^=o(l). 


and because ^ ^ = o(1)j the required budget is 5 = O (n(6 -I- Ta^pe^)') = 

□ 
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C Proof of Theorem |7] 


Theorem 7. Let ^ € (0,1), and t > 1. Let || • || denote the spectral norm. If {xi}i^[n] ore i.i.d. and sampled uniformly 
from the unit ball, then with probability at least 1 — d~^ , when n > C'(|)^(d + 2) log d, for some absolute constant 


C, then, 


1 


T v\-l| 


X'X < (1 + 0-, — 7:n,and (X ' X) 

I II - V + 2 ' 


< 




and 


yl + X' X\\ <7+11+0-;- n, and \\hl + X ' X)-'-\\ < 

II - ' V ^^d + 2 ^ II - 


7 + ( 1-0 


d+2 


Proof We will first require LemmafTTl which characterizes the covariance matrix of the distribution on X. 

Lemma 11. The covariance matrix ofx is S = '3^^- 

Proof Let zi,... ,Zd ^ -1V(0,1), and let u ^ U[0, 1], all drawn independently. Define, r = \/zf + ■ • ■ + and 
Z = Then Z describes a uniform distribution over the d-dimensional unit ball ifisl] . Recall that 

this is the same distribution from which the Xi are drawn. By the symmetry of the uniform distribution, E[Z] = 0, 
and Cov{Z) must be some scalar times the Identity matrix. Then to compute the covariance matrix of Z, it will 
suffice to compute the variance of some coordinate Zi of Z. Since each coordinate of Z has mean 0, then Var(Zi) = 
E[Zf]+E[Z,]^ =E[Zf]. 




= E 


£7 

Y. (“‘"7)' 


= E[m2/‘^]E 

= E[m2/<^] 

d 

d + 2 




By symmetry of coordinates, E[Zj^] = E[Z^] for all i, j. Then E[Zj^] = and the covariance matrix of Z (and of 
the Xi since both variables have the same distribution) is E = 


□ 


From Corollary 5.52 in ll2-5ll and the calculation of covariance in LemmafTTl for any f G (0,1) and < > 1, with 
probability at least 1 — d~* , 


-X^X - 
n d + 2 


<e 


d + 2’ 


(7) 


when n > (7(1)^ (d + 2) logd, for some absolute constant C. We assume for the remainder of the proof that inequality 
O holds, which is the case except with probability at most d”* , as long as n is sufficiently large. Then 


- 


d + 2 


-nl 


<S, 


d + 2 


Let Ainax(^) and Ainin(^) denote respectively the maximum and minimum eigenvalues of a matrix A. By defini¬ 
tion, Aniax(^) — ll^ll- 
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Assume towards a contradiction that Aniax(-^^-^) = (1 + C) ^ for (5 > 0. 


’d + 2 


n > 


a:'a:- 


d + 2 


nl 


= at'a: - 


d + 2 


= Xm^AX^X)- 


d + 2 


+ + JT2" 


= « 


1 


d + 2 


This implies i5 < 0, which is a contradiction. Thus Aniax(-Ar^X) = ||X^Ar|| < (1 + 

Similarly, assume that Aniin(-^^-^) = (1 ~ C) ~ ^ fo^' some 5 > 0. Since all eigenvalues are positive, it must 
be the case that Aniin(-Y^-^) > 0. 

0 > X^iniX^X --^nl) 


= Xnun(X^X)- 


d + 2 
1 

- 1 

d + 2 


= (1 — g) ^ n — 6 — 

d+2 d+2 


= 


1 


d + 2 


n — S 


This is also a contradiction, so Aniin(-^^-^) > (1 — For any matrix A, Aniax(^ = i+T 3T- Thus, 


-^min (^) 


A„,i„(X^X) = 


1 


A„,ax ((X^X)-!) 
1 


ll(XTX)- 


> ( 1-0 

ll(XTX)-lll < (1-c) 


1 


d + 2 


d + 2 


Using the fact that A is an eigenvalue of a matrix A if and only if (A + c) is an eigenvalue of {A + cl), we have 
the following inequalities to complete the proof: 


\ll + ^^^11 — ^max(7f + X^X) < 7 + (1 + 0 


1 


d + 2 


UjI + X'X)-^\\ = 


< 


1 


X^UlI + X^X) - ^ + (1 - ^) 


d+2 


□ 


D Quadratically Bounded Privacy Penalty Costs 

We will consider a particular functional form of fi{ci,e), motivated by the model of privacy cost in the existing 
literature |@]. In particular, we assume that each player additionally has a privacy cost function gi that measures her 
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loss for participating in a particular instantiation of a mechanism. Further, we assume that gi is upper-bounded by a 
function that depends on the effect that player z’s report has on the mechanism’s output. This assumption leverages the 
functional relationship between player i’s data {xi, yi), and the output of the mechanism. For example, if a particular 
mechanism ignores the input from player i, then her privacy cost should be 0 for participating in that computation, 
since her data is not used. We then define her ex ante privacy cost /i(ci, e) to be her expected cost for participation, 
where the expectation is taken over the randomness of other players’ data and reports. 

To formally state this assumption, first let mechanism take in data reports {X, y) and output an estimated 
parameter 0. Define gi{M, 9, {xi, yi), (X_i, y-i)) to be the privacy cost to player i for reporting {xi, yi) to mechanism 
Ai when all other players report {X-i, y-i) and the output of Ai is 9. 

Assumption 4 (|@1, Privacy Cost Assumption)]! We assume that for any mechanism M that takes in data {X, y) and 
outputs an estimate 9, then for all players i,for all estimates 9, and for all possible input data {X, y). 


gi{M,9, {Xi,yi), {X-i, y-i)) < Ciln max 


Pr[M{X,y{,y-i) = 9] 


y'.y'' Pr[M{X,y'l,y-i)=9] 


Lemma 12 (ifl^l^, Composition Lemma). In settings that satisfy Assumption^and for mechanisms M that are e- 
differentially private for e < 1, then for all players i with data {xi, yf), for all data reports of other players {X-i, y-i), 
and for all possible misreports y{ by player i. 


E[gi{M,M{X,y), {xi,yi), {X-i, y-i))] - E[gi{M,M{X,y{,y-i), {xi,yi), {X-i, y-i))] < 2cit{e^ - 1) < 4cie^ 

Proof (Sketch) The first inequality comes from Lemma 5.2 of |@] by plugging in our specification of their “privacy- 
bound function” and replacing statistical difference with the upper bound of — 1. The second inequality comes from 
the bound < 1 -f 2e for small e. □ 


To combine this framework with the utility model introduced in Section |2j4] we need only to interpret fi{ci, e) = 
^E[gi{M, M{X, y), {xi,yi), {X-i, y-i))]. That is, f{ci, e) is player Fs expected cost for participating in the mecha¬ 
nism (up to a scaling constant). This interpretation, along with Lemma[T2] motivates Assumption!!] 


E Strong Convexity of Regularized Loss 


Recall that we consider the loss function £{9, X, y) to be the sum of these individual loss functions plus a regularizing 
term: 

n n 

£{ 9 - X,y)=^ f(6»; Xi,yi) = - 9 ^ Xi)"^ + 7 ||6»||2 • 

i=l i=l 

We now define strong convexity, which requires that the eigenvalues of the Hessian of a function are bounded away 
from zero, and we prove that the loss function £ is strongly convex. 

Definition 10 (Strong Convexity). A function f ^ R is m-strongly convex if 


H (/(x)) ~ tnl is positive semi-definite for all x C 


where H{f{x)) is the Hessiat^of f, and I is the d x d identity matrix. 

^The assumption proposed in allows privacy costs to be bounded by an arbitrary function of the log probability ratio that satisfies certain 
natural properties. We restrict to this particular functional form for simplicity, following m. 

®The Hessian H of function / is a d X d matrix of its partial second derivatives, where 




dxjdxk 


Ad X d matrix A is positive semi-definite (PSD) if for all v S R'*, Av > 0. 
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Notice that when / is a one-dimensional function (d = 1), strong convexity reduces to the requirement that 
fix) > m > 0 for all x G R- The following lemma proves that regularizing the quadratic loss £ ensures that it is 
strongly convex. 

Lemma 13. C{0; X, y) is 2'^-strongly convex in 0. 

Proof. We first compute the Hessian of £{0; X, y). For notational ease, we will suppress the dependence of £ on X 
and y, and denote the loss function as £{0). We will use Xij to denote the j-th coordinate of Xi, and 0j to denote the 
j-th coordinate of 0. 

QQ — 'y -f 2(0 Xi)xij'\ + 2'y0j 

^ i—1 

wyik = g|2(i,/.)*«l for; 

3 i—1 

The Hessian of £ is, 

n 

Hi£{0)) = J2xixJ + f I, 

i=l 

where / is the identity matrix. Thus, 

n 

H{£{0)) - 27 / = '^XixJ, 

which is positive semi-definite. To see this, let v be an arbitrary vector in Then for each i, v{xixj = {vxf 
0. The sum of PSD matrices is also PSD, so £{0) is 27 -strongly convex. 
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□ IV 



